home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Internet Info 1994 March
/
Internet Info CD-ROM (Walnut Creek) (March 1994).iso
/
inet
/
internet-drafts
/
draft-ietf-osids-dsa-metrics-00.txt
< prev
next >
Wrap
Text File
|
1993-03-03
|
36KB
|
1,109 lines
Network Working P. Barker and S.E. Hardcastle-Kille
Group University College London
INTERNET-DRAFT September 1992
DSA Metrics
(OSI-DS 34 (v2))
Status of this Memo
This document defines a set of criteria by which a DSA implementation
may be judged. Particular issues covered include conformance to
standards; performance; demonstrated interoperability.
The intention is that the replies to the questions posed provide a
fairly full description of a DSA. Some of the questions will yield
answers which are purely descriptive; others, however, are intended to
elicit answers which give some measure of the utility of the DSA. The
marks awarded for a DSA in each particular area should give a good
indication of the DSA's capabilities, and its suitability for
particular uses.
Please send comments to the authors or to the discussion group
<osi-ds@CS.UCL.AC.UK>.
This document is an Internet Draft. Internet Drafts are working
documents of the Internet Engineering Task Force (IETF), its Areas,
and its Working Groups. Note that other groups may also distribute
working documents as Internet Drafts.
Internet Drafts are draft documents valid for a maximum of six months.
Internet Drafts may be updated, replaced, or obsoleted by other
documents at any time. It is not appropriate to use Internet Drafts
as reference material or to cite them other than as a "working draft"
or "work in progress."
Please check the I-D abstract listing contained in each Internet Draft
directory to learn the current status of this or any other Internet
Draft.
INTERNET--DRAFT DSA Metrics September 1992
Contents
1 Overview 2
2 General Information 3
3 Conformance to OSI Standards 4
3.1 Directory protocols................................ 4
3.2 Implementors' agreements and profiles ............ 5
3.3 Protocol stacks.................................... 6
3.4 Schema ............................................ 6
3.5 DIT structure .................................... 7
4 Conformance to Research Community Standards 7
5 Performance 7
5.1 Environment used for benchmarking ................ 9
5.2 Speed for various operations ..................... 9
5.2.1 Bind ........................................ 9
5.2.2 List ........................................ 10
5.2.3 Search ..................................... 10
5.2.4 Read ........................................ 11
5.2.5 Add entry.................................... 11
5.2.6 Modify entry ................................ 12
5.2.7 Modify RDN ................................. 12
5.2.8 Query rate ................................. 12
5.3 The results........................................ 13
6 Miscellaneous characteristics 13
7 Support for replication 14
8 Support for access control 15
9 Support for schema management 16
Barker and Hardcastle-Kille Expires 23 March 1993 Page 1
INTERNET--DRAFT DSA Metrics September 1992
10 Management tools 16
10.1 Dynamic system management ........................ 16
10.2 Static system management ......................... 16
10.3 Data management.................................... 17
11 Operational Use 17
12 Interoperability 17
1 Overview
The purpose of this document is to define some metrics by which DSA
products can be measured. Such metrics are valuable as whilst an
X.500 DSA must conform to the specification in the standard - this is
a sine qua non - protocol conformance is not in itself the hallmark of
a usable implementation. A DSA must perform operations within a
reasonable time; a DSA must offer good throughput of queries; a DSA
must be able to handle a reasonable volume of data; if modification
operations are provided, some sort of access control must be provided;
a DSA and its data must be manageable.
In many respects, it is almost impossible to say that one DSA is
better than other from looking at the responses to question in this
document. For some, the cost and level of support will be the key
criterion. For another user, the flexibility of the schema management
facilities, or the feasibility of running the DSA over an existing
relational database, will be of prime importance. In many respects
DSAs will just be different, rather than better or worse.
However, all other things being equal, the look-up speed of a DSA is
very obviously measurable, and there is a substantial number of
question on the speed of the various X.500 operations, and in
particular on the look-up operations.
Throughout this document, some of the questions posed are annotated
with a square-bracketed points score and an explanation as to how the
points should be allocated. For example, a question might be appended
with ``[2 if yes]'', indicating score 2 points for an affirmative
answer to that question. These points scores should be collated in
Table 1 at the end of the document. The questions on DSA performance
are judged to be important enough to have a separate table for those
results: they appear in Table 2. Together, these tables constitute a
measure of the DSA. The metrics are on a section by section basis,
Barker and Hardcastle-Kille Expires 23 March 1993 Page 2
INTERNET--DRAFT DSA Metrics September 1992
which should help the reader who is seeking, for example, a DSA with
fast look-up capabilities and extensive access control facilities, to
focus on the critical aspects of a DSA for their particular
requirement.
2 General Information
This section contains general information about the implementation
under discussion.
1. Name of the implementation ......................................
2. Version number of the DSA described in this document ............
3. Are there plans to continue development of this implementation?
[3 if yes] ......................................................
4. Name and address of supplier or person to contact ...............
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
5. Describe the hardware and software platforms supported by the DSA
[up to 4 points may be awarded for this question]
(a) Hardware (If appropriate, can summarise as, for example
``generic UNIX platform'') ..................................
(b) O/S (state version if critical)
i. UNIX) (be sure to indicate which flavour - e.g. SYSV [1],
BSD [1], SUNOS, etc) ....................................
ii. VMS) [1] ................................................
iii. MS-DOS [1] ..............................................
iv. Macintosh [1] ...........................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 3
INTERNET--DRAFT DSA Metrics September 1992
v. Other) [1] ..............................................
6. Name any other software required to run the system which is not
supplied with the operating system or with the DSA software
itself. Examples might include a database package, or
communications software .........................................
7. Is the software free? If the DSA needs other packages, are these
also freely available? [3 if completely free] ..................
....................................................................
3 Conformance to OSI Standards
3.1 Directory protocols
8. Does the DSA implement DAP? [2] .................................
9. Does the DSA implement DSP? [2] .................................
10. Does the implementation meet the conformance clauses in section
9.2 of X.519? [1 for yes]
Statement requirements
(a)...............................................................
(b)...............................................................
(c)...............................................................
(d)...............................................................
(e)...............................................................
(f)...............................................................
Static requirements [1 for yes]
(a)...............................................................
(b)...............................................................
(c)...............................................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 4
INTERNET--DRAFT DSA Metrics September 1992
(d)...............................................................
(e)...............................................................
(f)...............................................................
Dynamic requirements [1 for yes]
(a)...............................................................
(b)...............................................................
(c)...............................................................
(d)...............................................................
(e)...............................................................
11. Please list all conformance testing work applied to the
implementation ..................................................
....................................................................
....................................................................
....................................................................
....................................................................
3.2 Implementors' agreements and profiles
Does the DSA conform to the following implementors' agreements? If
so, state which version numbers.
12. EWOS? [1] .......................................................
13. OIW? [1] ........................................................
Does the DSA conform to the following profiles? If so, state which
version numbers.
14. UK GOSIP? [1] ...................................................
15. US GOSIP? [1] ...................................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 5
INTERNET--DRAFT DSA Metrics September 1992
State any other GOSIP profiles to which the DSA conforms ............
.....................................................................
3.3 Protocol stacks
For the next two questions, [2 per stack supported for up to 4 stacks]
16. Which of the following transport and network layer protocols does
the DSA support:
(a) TP.x over CONS (state transport class) ......................
(b) TP.4 over CLNS ..............................................
17. Does the DSA support other transport and ``network'' layer
protocols?
(a) TP.x over RFC1006 over TCP/IP (state transport class) .......
(b) TP.x over X.25(1980) (state transport class) ................
(c) State any other options supported. .........................
................................................................
18. Does the DSA also run over any lightweight stack? If so, describe
it with reference to the OSI seven layer model [3] ..............
....................................................................
3.4 Schema
19. Does the DSA support the full schema in X.520 and X.521, with
respect to the following? State any omissions.
(a) All object classes [1] ......................................
(b) All attribute types [1] .....................................
(c) All attribute syntaxes [1] ..................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 6
INTERNET--DRAFT DSA Metrics September 1992
3.5 DIT structure
20. A suggested DIT structure, detailing an object class hierarchy, is
presented in X.521. Does the DSA:
(a) Enforce this hierarchy? ....................................
(b) Allow the enforcement of this hierarchy? ...................
4 Conformance to Research Community Standards
The COSINE and Internet Directory Pilots have agreed a set of
extensions to the standard, which make for a more cohesive pilot.
This section is about conformance to these extensions.
21. Does the DSA fully support RFC1274, ``The COSINE and Internet
X.500 Schema''? [2] ............................................
If not, please supply a list of all those object classes and
attribute types in RFC1274 which are supported on a separate
sheet.
22. Does the DSA support RFC1276, ``Replication and Distributed
Operations extensions to provide an Internet Directory using
X.500''? [2] ...................................................
....................................................................
23. If the DSA uses RFC1006 at the network layer, does the DSA conform
to RFC1277, ``Encoding Network Addresses to support operation over
non-OSI lower layers'' [3] ......................................
24. If the DSA uses X.25(1980) at the network layer, does the DSA
conform to RFC1277, ``Encoding Network Addresses to support
operation over non-OSI lower layers'' [3] .......................
5 Performance
This section should give an outline to the expected performance of the
DSA. A number of operations are timed in order to give a feel for the
DSA's speed and throughput. Note that all operations should be
resolvable within a single DSA. Chaining and referral are not
assessed, although it should be possible to infer, albeit
Barker and Hardcastle-Kille Expires 23 March 1993 Page 7
INTERNET--DRAFT DSA Metrics September 1992
approximately, the speed of distributed operations.
i. The tests should be made against an organisational database of
20000 entries. Some tests are against subsets of this data, and
so the database should be set up according to the following
instructions.
Create an organisational DSA with 20000 entries below the
organisation node. Sub-divide this data into a number of
organisational units, one of which should contain 1000 entries,
another of which should contain 100 entries, and a third which
should contain just 10 entries. The entries, which should differ,
should be created with the following attributes:
(a) Common Name
(b) Surname
(c) Telephone number
(d) Postal Address (of 100 characters)
(e) Object class
ii. In all the tests, two timings should be taken. In order to
normalise the test results as much as possible, it is suggested
that these tests be undertaken on an otherwise lightly loaded
machine.
(a) A typical ``cold start'' reading should be given. In this
case the system will not have the advantage of any benefits
that derive from operating system paging, or caching.
(b) A best possible figure should be given, which indicates the
upper limit of DSA performance.
iii. The timings should relate to the default set-up. If significant
performance gains can be made by use of configuration options,
such as building extra indexes to support searches, measures of
the improved performance may also be given.
Attention should be also drawn to any optimisations, heuristic or
otherwise, which are not evidenced in the following tests.
Barker and Hardcastle-Kille Expires 23 March 1993 Page 8
INTERNET--DRAFT DSA Metrics September 1992
5.1 Environment used for benchmarking
The results will be directly correlated to the test set-up used, and
in particular, the hardware. Please answer the following questions to
describe the test environment:
(a) Processor (make and model) ..................................
(b) Processor speed (MIPS) ......................................
(c) Primary memory available ....................................
(d) O/S version .................................................
(e) Network type and bandwidth (e.g. 10 Mbit Ethernet) .........
(f) Protocols in transport layer and below (e.g. TP 0, RFC1006,
TCP/IP) .....................................................
(g) How/where timings obtained?
o C procedural interface ..................................
o DUA shell (e.g. Quipu's DISH) ..........................
Please note that the tests should be made using a DUA and DSA with
full 7-layer stacks, rather than some lightweight protocol.
5.2 Speed for various operations
The tests are described, one subsection per operation. The results
should be entered in Table 2 which follows the test descriptions.
5.2.1 Bind
The time it takes for a DUA to bind to the Directory. This time
should include all the initialisation time a DUA process needs before
it can query the Directory: e.g. reading of tailor files, schema
information, etc.
Give the bind time for each of the following levels of authentication.
State ``n/a'' if the implementation does not support a particular
Barker and Hardcastle-Kille Expires 23 March 1993 Page 9
INTERNET--DRAFT DSA Metrics September 1992
level of authentication.
25. Anonymous
26. Simple
27. Simple protected
28. Strong
5.2.2 List
Give the time for listing a set of organisational unit sibling
entries.
29. 10 entries
30. 100 entries
31. 1000 entries
5.2.3 Search
In this section, two sets of search operations should be performed on
the DSA.
i. A single level search of 100 entries within an organisational
unit.
ii. An organisation subtree search, on the subtree of 20000 entries.
The following searches should be tried. Unless otherwise stated, the
``XXX'' or ``YYY'' part of the search filter should be chosen in such
a way as to return a single result. Unless stated otherwise the
results should return all attributes for the entry.
32. Exact match for a surname:
surname=XXX
Barker and Hardcastle-Kille Expires 23 March 1993 Page 10
INTERNET--DRAFT DSA Metrics September 1992
33. Leading substring match for a common name:
commonName=XXX*
34. Any substring match for a common name:
commonName=*XXX*
35. Trailing substring match for a common name:
commonName=*XXX
36. Approximate match for a common name:
commonName"=XXX
37. More complex filter, searching by object class and two other
attribute types:
objectClass=person AND
(commonName=XXX* OR telephoneNumber=*YYY)
38. Search returning all entries (i.e. 100 entries in the single
level search, and all 20000 entries in the subtree search:
objectClass=*
In this case, no attribute values should be returned in the result
set.
5.2.4 Read
39. A single read operation, returning all attributes.
5.2.5 Add entry
40. Add an entry beneath an entry which has:
(a) 0 children
(b) 10 children
(c) 100 children
Barker and Hardcastle-Kille Expires 23 March 1993 Page 11
INTERNET--DRAFT DSA Metrics September 1992
(d) 1000 children
5.2.6 Modify entry
Modify an attribute value, other than an RDN value, for an entry which
has
1. 10 siblings
2. 100 siblings
3. 1000 siblings
41. Modify an entry
(a) Add description attribute
(b) Remove description attribute
5.2.7 Modify RDN
Modify an RDN value for an entry with the following number of
siblings.
42. Modify RDN
(a) 10 siblings
(b) 100 siblings
(c) 1000 siblings
5.2.8 Query rate
As the time taken for a single read will usually be negligible, the
following search and set of reads should give a clearer indication of
the query rate.
43. A single level search of the DIT, to return 100 entries for
persons, and then a read of each entry, returning just the surname
Barker and Hardcastle-Kille Expires 23 March 1993 Page 12
INTERNET--DRAFT DSA Metrics September 1992
attribute for each entry.
5.3 The results
The results of the tests just described should be entered in Table 2,
at the end of the document.
6 Miscellaneous characteristics
44. Does the DSA use its own database, or can it be used in
conjunction with a general-purpose database package such as
Oracle? [1 for own, 1 for ability to map onto general purpose
databases, 1 if any such mappings have been made] ...............
....................................................................
45. If the DSA runs as a static server, state the start-up time for a
DSA with a database of 20000 entries. If this varies widely
according to configuration options, give figures for the various
options. .......................................................
....................................................................
46. What is the maximum number of simultaneous associations that the
DSA may have open? [1 if more than associations] ...............
47. Maximum database size, in entries, megabytes, or as appropriate.
If none, state what the constraints are. [1 if a database of more
than 100,000 entries is feasible] ...............................
48. What use does the DSA make of indexing [2 if yes] ?
(a) Can the database be fully inverted? [1] ....................
If not, state for which attributes:
i. indexes are automatically built
ii. indexes may be built
49. What is the run-time size of an entry as specified in the previous
section on performance? (This should be the marginal size of an
entry and thus should include the overhead of indexes, etc.) ...
Barker and Hardcastle-Kille Expires 23 March 1993 Page 13
INTERNET--DRAFT DSA Metrics September 1992
50. What is the on-disk database size of an entry as specified in the
previous section on performance? ...............................
51. What sort of approximate match algorithm does the DSA use?
Describe it briefly .............................................
....................................................................
....................................................................
....................................................................
52. Does the DSA attempt to use relay DSAs (which have access to more
than one network) in order to achieve connectivity with DSAs which
are not on the same network? [2] ...............................
7 Support for replication
53. Does the DSA support the replication mechanisms as described in
the 1992 standard [2]?
....................................................................
54. Does the DSA support any other replication mechanisms? .........
(a) RFC1276 [2] .................................................
(b) Other (please give a reference to any description of the
mechanisms, and indicate whether these mechanisms are used by
any other implementations) [1 for any mechanism] ............
................................................................
................................................................
................................................................
55. If the DSA supports replication, does it support:
(a) Replication of a single entry? [2] .........................
(b) Replication of a set of sibling entries? [2] ...............
(c) Replication of a subtree? [2] ..............................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 14
INTERNET--DRAFT DSA Metrics September 1992
8 Support for access control
56. Does the DSA support access control as described in the 1992
standard [3]? ..................................................
57. Does the DSA have any access control mechanisms at all? [2] ....
58. If yes, does the access control scheme support the following:
(a) Allow a user to maintain their own entry? [1] ..............
(b) Allow a user to maintain some attributes in their own entry,
but not all attributes? [1] ................................
(c) Give management rights to a DSA manager in a fashion analogous
to the privileges given to a UNIX super-user? [1] ..........
(d) Give management rights to a data manager on a per subtree
basis? [1] .................................................
(e) Give management rights (to an entry, group of entries,
subtree, etc) to a group of users? [1] .....................
(f) Give access rights to users on the basis of the leading
portion of their Distinguished Name? [1] ...................
59. If there are features of the access control mechanisms which are
not brought out by the above questions, please describe these
additional features [up to 2 for wonderful additional features!]
....................................................................
....................................................................
....................................................................
....................................................................
60. Does the DSA support the extended access control techniques
described in ``An Access Control approach for Searching and
Listing'' by Hardcastle-Kille and Howes, in the Internet Draft,
OSI-DS 21. [2]
....................................................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 15
INTERNET--DRAFT DSA Metrics September 1992
9 Support for schema management
61. Does the DSA implement the schema management defined in the 1992
standard? [2] ..................................................
62. If not, is the schema stored in the Directory? [2] .............
63. Can a DSA manager extend the schema and add new
(a) Attribute types with existing syntaxes? With compilation [1],
or without compilation [2] ..................................
(b) Attribute sets? With compilation [1], or without compilation
[2] .........................................................
................................................................
(c) Object classes? With compilation [1], or without compilation
[2] .........................................................
................................................................
(d) Attribute syntaxes? With compilation [1], or without
compilation [2] .............................................
64. Is it possible to add in or modify DIT structure rules, with
compilation [1], without compilation [2] ........................
10 Management tools
10.1 Dynamic system management
65. Are there tools for monitoring DSA activity? [1] ...............
66. Are there tools for controlling a run-time DSA? [2] .............
10.2 Static system management
67. If knowledge is stored within the DIT, are there tools for
knowledge management? [1] ......................................
68. Are there tools for checking that attributes with Distinguished
Name syntax contain values of entries in the DIT (i.e. they do
Barker and Hardcastle-Kille Expires 23 March 1993 Page 16
INTERNET--DRAFT DSA Metrics September 1992
not contain ``dangling pointers'')? [1] ........................
10.3 Data management
69. If the DSA doesn't use a general-purpose database package, what
data management tools are available? ...........................
....................................................................
11 Operational Use
The DSA may have lots of wonderful features -- on paper! But has the
DSA been shown to work in practice? The following measures are
intended to give some measure of confidence that the DSA's viability
has been demonstrated.
70. How many entries in the largest DSA in use in operational use? :
71. What is the largest set of DSAs supporting an organisation? ....
72. What is the estimated number of organisations using this
implementation for service use? [8 if more than 100
organisations, 5 if more than 50 organisations, 3 if more than 20
organisations, 2 if more than 5 organisations, 1 if more than 1
organisation] ...................................................
73. Is this DSA used commercially with an installed base of more than
10 customers? [2] ..............................................
12 Interoperability
The X.500 Directory is the OSI Directory. OSI stands for Open Systems
Interconnection -- DSAs have to be able to inter-operate. They also
have to be seen to interoperate.
74. Is this DSA in use in X.500 pilots? ............................
(a) Is this DSA in use anywhere in the COSINE/Internet Pilot? [3]
................................................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 17
INTERNET--DRAFT DSA Metrics September 1992
(b) Is this DSA in use in any other major pilot? [2] ...........
75. Name any other systems which you believe the system to
interoperate with. (It is not sufficient to say ``any system
which supports the conformance clauses ...)'' ..................
76. Name any systems which have been publicly demonstrated to
interwork with the DSA [1 per implementation, up to maximum of 5]
....................................................................
....................................................................
....................................................................
....................................................................
....................................................................
Barker and Hardcastle-Kille Expires 23 March 1993 Page 18
INTERNET--DRAFT DSA Metrics September 1992
_______________________________________________
|____________Section__________||____Points_____ |
|_No._|Description_____________|Maximum_|Scored_|_
| | | | |
|___2_|General_Information_____|__10___|:_...__|
| | | | |
|___3_|Conformance_to_OSI______|__25___|:_...__|
| |Conformance to Research | | |
|___4_|Community_standards_____|__10___|:_...__|
| | | see | |
|___5_|Performance_____________|table_2_|....__|
| | | | |
|___6_|Miscellaneous___________|__10___|:_...__|
| | | | |
|___7_|Replication_____________|__10___|:_...__|
| | | | |
|___8_|Access_control__________|__15___|:_...__|
| | | | |
|___9_|Schema_Management_______|__12___|:_...__|
| | | | |
|__10_|Management_tools________|__5____|:_...__|
| | | | |
|__11_|Operational_use_________|__10___|:_...__|
| | | | |
|__12_|Interoperability________|__10___|:_...__|
Table 1: DSA Metrics
Barker and Hardcastle-Kille Expires 23 March 1993 Page 19
INTERNET--DRAFT DSA Metrics September 1992
______________________________________________________
|Operation || Cold DSA || Optimum |
|__________________||_______________||___Performance__|_
|Bind || || |
| --Anonymous || ..............|| ............. |
| --Simple || ..............|| ............. |
| --Simple Prot || ..............|| ............. |
|___--Strong_______||_..._...._..._:||_..._...._....__|
|List || || |
| -- 10 entries || ..............|| ............. |
| -- 100 entries || ..............|| ............. |
|___--_1000_entries||_..._...._..._:||_..._...._....__|
|Search |single |subtree |single |subtree |
| _|level_|________|_level_|________|
| | | | | |
| -- exact |..... | ...... | ..... | ...... |
| -- leading sub |..... | ...... | ..... | ...... |
| -- any sub |..... | ...... | ..... | ...... |
| -- trailing sub |..... | ...... | ..... | ...... |
| -- approx |..... | ...... | ..... | ...... |
| -- complex |..... | ...... | ..... | ...... |
|___--_return_all___|..._:_|_...._:_|_..._:_|_...._:_|_
| || || |
|Read______________||_..._...._..._:||_..._...._....__|
|Add || || |
| 0 siblings || ..............|| ............. |
| 10 siblings || ..............|| ............. |
| 100 siblings || ..............|| ............. |
|____1000_siblings_||_..._...._..._:||_..._...._....__|
| || || |
|Modify || || |
| 10 siblings || ..............|| ............. |
| 100 siblings || ..............|| ............. |
|____1000_siblings_||_..._...._..._:||_..._...._....__|
| || || |
|Modify RDN || || |
| 10 siblings || ..............|| ............. |
| 100 siblings || ..............|| ............. |
|____1000_siblings_||_..._...._..._:||_..._...._....__|
| || || |
|Query_rate________||_..._...._..._:||_..._...._....__|
Table 2: Speed of operations
Barker and Hardcastle-Kille Expires 23 March 1993 Page 20